Composition control device, composition control method, and program

ABSTRACT

Operability is improved for an operation related to composition adjustment.Composition designation operation information is acquired that is operation information designating a composition of an imaging device and includes information of a designated position on a screen that displays a captured image of the imaging device, and an imaging range of the imaging device is controlled to adjust the composition of the imaging device on the basis of subject information corresponding to the designated position of the composition designation operation information. Thus, the adjustment to the target composition is performed on the basis of the subject information corresponding to the designated position on the screen displaying the captured image.

TECHNICAL FIELD

The present technology relates to a composition control device, acomposition control method, and a program, and particularly relates to atechnology related to reception of operation in designating a targetcomposition.

BACKGROUND ART

For example, a platform device capable of driving an imaging device soas to change an imaging direction, such as an electric gimbal, is known.Then, the subject tracking can be performed using this type of platformdevice. Specifically, control of the platform device (drive control ofan actuator built in the platform device) is performed such that atarget subject is positioned at a predetermined position such as acenter position in a captured image.

Patent Document 1 below discloses a subject tracking technique using theplatform device. Specifically, in Patent Document 1, first, the user iscaused to execute an operation of panning and tilting the platformdevice, so as to arrange a subject to be tracked at a desired positionin the screen. That is, adjustment to a target composition at the timeof tracking is performed on the basis of the pan and tilt operation ofthe user. Then, when the operation button 10 is operated by the userafter the composition is adjusted in this way, the tracking operation ofthe target subject is started so that the adjusted composition ismaintained.

Furthermore, in Patent Document 1, the composition adjusted by the panand tilt operation as described above can be registered as the targetcomposition at the time of tracking. Information of the registeredcomposition (specifically, information of center coordinates Mp and asize Ft of the tracking target frame for the tracking subject S) isassociated with a shot button 16, and when the shot button 16 isoperated, the tracking operation of the subject is started so as tomaintain the registered composition.

CITATION LIST Patent Document Patent Document 1: Japanese PatentApplication Laid-Open No. 2013-247508 SUMMARY OF THE INVENTION Problemsto be Solved by the Invention

In the technique described in Patent Document 1 above, since theadjustment to the target composition is performed on the basis of thepan and tilt operation of the platform device by the user, it isdifficult to adjust the composition to an intended composition in asituation where the subject moves. Furthermore, even in a case where thecomposition is switched to the pre-registered composition using the shotbutton 16, there is no difference in that the composition is adjusted inadvance by panning and tilting operations, and an operation foradjustment to the target composition is difficult.

The present technology has been made in view of the circumstancesdescribed above, and has an object to improve operability of anoperation related to composition adjustment.

Solutions to Problems

A composition control device according to the present technologyincludes a control unit configured to acquire composition designationoperation information that is operation information designating acomposition of an imaging device and includes information of adesignated position on a screen that displays a captured image of theimaging device, and control an imaging range of the imaging device toadjust the composition of the imaging device on the basis of subjectinformation corresponding to the designated position of the compositiondesignation operation information.

The composition designation operation information includes informationof the designated position on the screen on which the captured image isdisplayed. Thus, adjustment to a target composition is performed on thebasis of an intuitive operation of designating a position on the screenfor displaying the captured image. Specifically, in the configurationdescribed above, the composition of the imaging device is adjusted onthe basis of the subject information corresponding to the designatedposition on the screen. The subject information is information regardinga subject in the captured image, and is at least information indicatingthe presence or absence of the detected subject. Thus, the compositionis adjusted on the basis of an intuitive operation such as an operationof designating the position of the detected subject on the screen.

The composition control device according to the present technologydescribed above can be configured such that in a case where the subjectinformation indicates a detected subject detected from the capturedimage, the control unit controls the imaging range in such a manner asto track the detected subject.

Thus, an operation of designating the subject to be tracked is anoperation of designating the detected subject in the screen.

The composition control device according to the present technologydescribed above can be configured such that in a case where the subjectinformation does not indicate the detected subject, the control unitdoes not perform control for tracking the subject at the designatedposition.

Thus, for example, in a case where a subject estimated as not desired bythe user to be tracked, such as a background portion of the detectedsubject, is designated, tracking of the subject at the designatedposition is not performed.

The composition control device according to the present technologydescribed above can be configured such that the composition designationoperation information includes direction designation operationinformation based on a direction designation operation that designates adirection in a plane of the screen, and the control unit controls theimaging range on the basis of the direction designation operationinformation.

With this direction designation operation, it is possible to designate adirection related to the composition adjustment such as a direction ofchanging a position of the subject designated by an operation ofdesignating a position in the screen.

The composition control device according to the present technologydescribed above can be configured such that the composition designationoperation information is detected by a touch operation on the screen,and the designated position is a touch start position of the screen.

Thus, an operation of designating the subject to be tracked can be asimple operation of touching the subject on the screen.

The composition control device according to the present technologydescribed above can be configured such that the direction designationoperation information is detected by an operation of tracing the screen.

Thus, the operation of designating a direction related to thecomposition adjustment can be achieved by an intuitive operation ofsliding a finger in a direction in which the composition is desired tobe changed on the screen.

The composition control device according to the present technologydescribed above can be configured such that the control unit controlsthe imaging range by controlling a platform device to which the imagingdevice is attached.

Thus, it is possible to adjust the composition by changing the imagingdirection of the imaging device by the platform device.

The composition control device according to the present technologydescribed above can be configured such that the platform device iscapable of adjusting a direction of the imaging device in a pan or tiltdirection, and the composition designation operation informationdesignating the pan or tilt direction is detected by an operation oflinearly tracing the screen.

Thus, the operation of designating the pan or tilt direction is achievedby an intuitive operation of linearly sliding the finger along the panor tilt direction.

The composition control device according to the present technologydescribed above can be configured such that the platform device iscapable of adjusting a direction of the imaging device in a rolldirection, and the composition designation operation informationdesignating the roll direction is an operation of tracing the screen inan arc shape.

Thus, an operation of designating the roll direction is achieved by anintuitive operation of sliding the finger in an arc shape along the rolldirection.

The composition control device according to the present technologydescribed above can be configured such that the control unit controlsthe imaging range by controlling a platform device to which the imagingdevice is attached, and controls the platform device to change adirection of the imaging device in a direction opposite to the directiondesignated by the direction designation operation in a case where thesubject information indicates a detected subject detected from thecaptured image, and to change the direction of the imaging device in thedirection designated by the direction designation operation in a casewhere the subject information does not indicate the detected subject.

In a case where a position designation operation is an operation ofdesignating the position of the detected subject, it can be estimatedthat the user desires to change the position of the detected subject inthe screen by a subsequent direction designation operation, and thus thedirection of the imaging device is changed in the direction opposite tothe direction designated by the direction designation operation. On theother hand, in a case where the position designation operation is anoperation of designating a position other than the detected subject, itcan be estimated that the user directly designates a changing directionof the imaging direction by the subsequent direction designationoperation, and thus the direction of the imaging device is changed inthe direction designated by the direction designation operation.

The composition control device according to the present technologydescribed above can be configured such that the control unit controlsthe imaging range by controlling a platform device to which the imagingdevice is attached, and controls, in a case where the subjectinformation indicates the detected subject, the platform device using aparameter corresponding to a type of the detected subject as a parameterrelated to the tracking.

Thus, it is possible to perform tracking control by a parametercorresponding to the designated subject, such as performing trackingcontrol by a parameter with increased followability in a case where thedesignated subject is a type of subject that moves quickly andperforming tracking control by a parameter with decreased followabilityin a case where the designated subject is a type of subject that movesslowly.

The composition control device according to the present technologydescribed above can be configured such that the platform device iscapable of adjusting directions of the imaging device in a pan directionand a tilt direction, and the control unit uses the parametercorresponding to the type of the detected subject for each of control inthe pan direction and control in the tilt direction of the platformdevice.

Thus, tracking characteristics in the pan direction and the tiltdirection can be individually set according to the type of thedesignated subject. It is possible that, for example, in a case wherethe subject is a bird, tracking responsiveness is increased in both thepan and tilt directions, and in a case where the subject is a dog, thetracking responsiveness in the pan direction is enhanced but thetracking responsiveness in the tilt direction is lowered.

The composition control device according to the present technologydescribed above can be configured such that the control unit performstracking control in one of a first tracking mode in which the parameterhaving high tracking responsiveness in the pan direction and the tiltdirection is used, a second tracking mode in which the parameter havinglow tracking responsiveness in the pan direction and the tilt directionis used, and a third tracking mode in which the parameter having lowtracking responsiveness in one of the pan direction and the tiltdirection and having high tracking responsiveness in the other of thepan direction and the tilt direction according to the type of thedetected subject.

Thus, it is possible to prevent tracking in which responsiveness isexcessively enhanced in a direction in which it is not necessary toenhance the tracking responsiveness according to the type of thedesignated subject.

The composition control device according to the present technologydescribed above can be configured such that the control unit adjusts,after control of the platform device using the parameter correspondingto the type of the detected subject is started, the parameter on thebasis of an error between a position of the detected subject and atarget position of tracking.

It is possible to determine whether or not the set parameter is suitablefor actual movement of the subject according to the magnitude of theerror between a designated position of the subject and the targetposition of tracking.

The composition control device according to the present technologydescribed above can be configured such that the control unit controls,in a case where the subject information indicates a detected subjectdetected as a person in the screen, the imaging range on the basis ofsimplified model information of a human body detected for the detectedsubject.

The simplified model information of the human body is informationindicating an arrangement state of a plurality of parts constituting thehuman body such as eyes, ears, a nose, a mouth, a neck, shoulders,elbows, wrists, buttocks, knees, and ankles in the human body. With theabove configuration, the control of the imaging range for changing thecomposition can be performed not on the basis of the operation ofdirectly giving an instruction on the driving direction or the driveamount of the camera platform or the focal length adjustment operation,but on the basis of a position and posture of each part of the subject,for example, estimated from the simplified model information describedabove.

The composition control device according to the present technologydescribed above can be configured such that the control unit controlsthe imaging range in such a manner that a center of an angle of view islocated on a front direction side of the detected subject on the basisof information of a direction of the detected subject estimated from thesimplified model information.

Thus, it is possible to adjust the composition to a natural compositionin which a space is secured on a front side of the designated subject.

The composition control device according to the present technologydescribed above can be configured such that the composition designationoperation information is detected by a touch operation on the screen,the designated position is a touch start position of the screen, and thecontrol unit controls the imaging range based on the simplified modelinformation on the basis of an operation on the screen after detectionof the designated position.

Thus, the composition adjustment based on the simplified modelinformation, such as adjustment to a composition in which the entirebody of the subject appears or a composition in which only a half bodyappears for example, can be achieved by a simple operation ofdesignating a target subject by a touch operation on the screen and thenperforming an operation on the screen.

The composition control device according to the present technologydescribed above can be configured such that the control unit controlsthe imaging range on the basis of a result of image recognitionprocessing performed on a captured image by an imaging device separatefrom the imaging device in which a captured image is displayed on thescreen.

Thus, as a captured image used for the image recognition processing, acaptured image with camera settings suitable for the image recognitionprocessing can be used instead of the captured image with camerasettings according to a drawing intention of the user.

Furthermore, a composition control method according to the presenttechnology is a composition control method including acquiringcomposition designation operation information that is operationinformation designating a composition of an imaging device and includesinformation of a designated position on a screen that displays acaptured image of the imaging device, and controlling an imaging rangeof the imaging device to adjust the composition of the imaging device onthe basis of subject information corresponding to the designatedposition of the composition designation operation information.

Even with such a composition control method, a similar operation to thatof the composition control device according to the present technologydescribed above can be obtained.

Furthermore, a program according to the present technology is a programcausing an information processing device to achieve a function includingacquiring composition designation operation information that isoperation information designating a composition of an imaging device andincludes information of a designated position on a screen that displaysa captured image of the imaging device, and controlling an imaging rangeof the imaging device to adjust the composition of the imaging device onthe basis of subject information corresponding to the designatedposition of the composition designation operation information.

With such a program, the composition control device according to thepresent technology described above is achieved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory view of an external appearance configurationexample of an imaging system including a composition control device as afirst embodiment according to the present technology.

FIG. 2 is a block diagram illustrating an internal configuration exampleof a platform device in the embodiment.

FIG. 3 is a block diagram illustrating an internal configuration exampleof the composition control device as the first embodiment.

FIG. 4 is a diagram for explaining composition adjustment in a casewhere a detected subject is designated.

FIG. 5 is a diagram for explaining, together with FIG. 4, compositionadjustment in a case where a detected subject is designated.

FIG. 6 is a diagram for explaining subject tracking control processingin the embodiment.

FIG. 7 is a diagram for explaining composition adjustment in a casewhere a subject other than the detected subject is designated.

FIG. 8 is a diagram for explaining composition adjustment together withFIG. 7 in a case where a subject other than the detected subject isdesignated.

FIG. 9 is a diagram for explaining an operation of changing an angle ofview.

FIG. 10 is a diagram for explaining the operation of changing the angleof view together with FIG. 9.

FIG. 11 is a diagram for explaining composition adjustment according toa double tap operation.

FIG. 12 is a diagram for explaining composition adjustment according toa double tap operation together with FIG. 11.

FIG. 13 is a flowchart illustrating an example of a specific processingprocedure for achieving a composition adjustment method as the firstembodiment.

FIG. 14 is a flowchart of normal drive mode processing according to theembodiment.

FIG. 15 is a flowchart of subject tracking mode processing according tothe embodiment.

FIG. 16 is a flowchart illustrating an example of a specific processingprocedure for achieving a composition adjustment method as a secondembodiment.

FIG. 17 is a flowchart illustrating an example of parameter settingprocessing according to a subject type.

FIG. 18 is a diagram for explaining an example of a type of a subject asa target in the parameter setting processing.

FIG. 19 is a flowchart illustrating an example of processing forparameter adjustment.

FIG. 20 is an explanatory diagram of simplified model information of ahuman body.

FIG. 21 is a diagram illustrating an example of a composition in whichswitching is performed on the basis of the simplified model information.

FIG. 22 is a diagram illustrating an example of a captured imagecorresponding to each composition illustrated in FIG. 21.

FIG. 23 is a flowchart illustrating an example of a specific processingprocedure for achieving a composition adjustment method as a firstexample in a third embodiment.

FIG. 24 is a flowchart illustrating processing content of step S501illustrated in FIG. 23.

FIG. 25 is a diagram for explaining a composition adjustment method as asecond example in the third embodiment.

FIG. 26 is a diagram for explaining a composition adjustment method asthe second example in the third embodiment together with FIG. 25.

FIG. 27 is a flowchart illustrating an example of a specific processingprocedure for achieving a composition adjustment method as the secondexample in the third embodiment.

FIG. 28 is a diagram illustrating a configuration of an imaging systemas a modification example.

FIG. 29 is a block diagram illustrating an internal configurationexample of each imaging device in a modification example.

FIG. 30 is a diagram exemplifying a platform device that can be drivenin a roll direction.

FIG. 31 is a diagram for explaining a modification example in which adirection of the imaging device in the roll direction is changed.

FIG. 32 is a diagram for explaining a modification example in which thedirection of the imaging device in the roll direction is changedtogether with FIG. 31.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments will be described in the following order.

1. First Embodiment

[1-1. Device configuration example][1-2. Composition adjustment][1-3. Processing procedure]

2. Second Embodiment 3. Third Embodiment

[3-1. First example][3-2. Second example]<4. Modification examples><5. Summary of embodiment><6. Present technology>

1. First Embodiment 1-1. Device Configuration Example

FIG. 1 is an explanatory view of an external appearance configurationexample of an imaging system including a composition control device(imaging device 1) as a first embodiment according to the presenttechnology.

The imaging system according to the present embodiment includes animaging device 1 and a platform device 50. In the imaging system, animaging direction of the imaging device 1 is changed by a rotationoperation of the platform device 50 in a state where the imaging device1 is mounted on the platform device 50. In particular, the platformdevice 50 includes an actuator as described later, and automatictracking is performed on a subject to be tracked by driving andcontrolling the actuator.

Note that the “imaging direction” is a direction corresponding to adirection in which the imaging device 1 performs imaging, and refers toa front direction (direction indicating a subject side) in an opticalaxis line of an imaging optical system included in the imaging device 1.In a case of the system as in FIG. 1, the imaging direction is changedby the rotation angle of the platform device 50, and thus the imagingdirection is uniquely determined by a rotation angle of the platformdevice 50.

FIG. 1A illustrates a state in which the imaging device 1 is mounted(attached) on the platform device 50, and FIG. 1B illustrates only theplatform device 50.

The platform device 50 is provided with a rotation shaft part 51 forrotation in a pan direction (in other words, a yaw direction) indicatedby an arrow D1 in the drawing and a rotation shaft part 52 for rotationin a tilt direction (in other words, a pitch direction) indicated by anarrow D2, and is provided with a base part 53, an attachment part 54,and an arm part 55.

The attachment part 54 is, for example, an L-shaped member, and a jointmechanism 54 a corresponding to a mechanism that is not illustrated andformed on a bottom portion of the imaging device 1 is provided on a topsurface of the bottom portion, by which the imaging device 1 can befixed as illustrated in FIG. 1A.

The attachment part 54 is attached to the arm part 55 via the rotationshaft part 52, so that the attachment part 54 is rotatable in the tiltdirection with respect to the arm part 55.

The arm part 55 is, for example, an L-shaped member, and is attached tothe base part 53 on the rotation shaft part 51 side. Thus, the arm part55 (and the attachment part 54 connected to the arm part 55) isrotatable in the pan direction.

For example, by using such a platform device 50, the imaging directionof the imaging device 1 can be displaced in the pan direction and thetilt direction. Thus, it is possible to change the composition of thecaptured image by the imaging device 1.

Here, changing the imaging direction of the imaging device 1 by theplatform device 50 is a kind of changing the imaging range of theimaging device 1.

In the imaging system illustrated in FIG. 1A, for example, it is assumedthat a handle part that is not illustrated is mounted on the base part53 of the platform device 50, and a user as an imaging person holds thehandle part in a hand to use the handle part. Alternatively, it is alsoassumed that the base part 53 is mounted on a pedestal such as a tripodand the user uses the platform device 50 without holding the platformdevice in the hand.

FIG. 2 is a block diagram illustrating an internal configuration exampleof the platform device 50.

The platform device 50 includes an actuator 56, a drive control unit 57,and a communication unit 58.

In this example, the actuator 56 is provided with a panning directionactuator (for example, a motor) that rotationally drives the rotationshaft part 51 and a tilting direction actuator (for example, a motor)that rotationally drives the rotation shaft part 52.

The drive control unit 57 includes a drive circuit of the actuator 56, acontrol circuit that controls the drive circuit, and the like, andperforms drive control of the actuator 56. In particular, the drivecontrol unit 57 of this example performs drive control of the actuator56 in accordance with information input via the communication unit 58.

The communication unit 58 performs data communication with an externaldevice according to a predetermined communication format. In particular,the communication unit 58 of this example performs data communicationaccording to a communication format corresponding to a communicationunit 11, which will be described later, included in the imaging device1.

FIG. 3 is a block diagram illustrating an internal configuration exampleof the imaging device 1.

The imaging device 1 is configured as a digital camera device, and canimage a subject, record image data as a moving image or a still image ona recording medium, or transmit the image data to an external device.

As illustrated, the imaging device 1 includes a lens system 2, animaging unit 3, a camera signal processing unit 4, an image recognitionprocessing unit 5, a recording control unit 6, a driver unit 7, acontrol unit 8, an operation unit 9, a display unit 10, a communicationunit 11, and a bus 12. The camera signal processing unit 4, the imagerecognition processing unit 5, the recording control unit 6, the controlunit 8, the display unit 10, and the communication unit 11 are connectedto the bus 12, and these units can perform data communication with eachother via the bus 12.

The lens system 2 includes lenses such as a cover lens, a zoom lens, anda focus lens, a diaphragm mechanism, and the like. Light (incidentlight) from a subject is guided by the lens system 2 and condensed on animaging surface of an imaging element in the imaging unit 3.

The imaging unit 3 includes, for example, an imaging element such as acomplementary metal oxide semiconductor (CMOS) type or a charge coupleddevice (CCD) type.

The imaging unit 3 performs, for example, correlated double sampling(CDS) processing, automatic gain control (AGC) processing, and the likeon an electric signal obtained by photoelectrically converting lightreceived by the imaging element, and further performs analog/digital(A/D) conversion processing. Then, the captured image signal as digitaldata is output to the camera signal processing unit 4 in the subsequentstage.

The camera signal processing unit 4 is configured as an image processingprocessor by, for example, a digital signal processor (DSP) or the like.The camera signal processing unit 4 performs various types of signalprocessing on a digital signal (captured image signal) from the imagingunit 3. For example, preprocessing, synchronization processing, YCgeneration processing, resolution conversion processing, and the likeare performed.

The image recognition processing unit 5 performs image analysis on thecaptured image signal input via the camera signal processing unit 4, andperforms recognition processing of image contents. In particular, in thecase of the present embodiment, the image recognition processing unit 5performs detection of a subject, detection of a face, and processing ofrecognizing the type of a detected subject in the frame of the capturedimage signal. Examples of the type of the subject recognized in theprocessing of recognizing the subject include a person, an animal (dog,cat, bird, and the like), an automobile, an airplane, and a train.

In this example, the image recognition processing unit 5 specifies a“position” and a “range” of the subject in processing of detecting thesubject. Here, as the “position” of the subject, an attention positionof the subject is specified. Here, the attention position means aposition to be noted determined for a target subject, and for example,in a case of a subject as a person, the position of interest can includea center (a center in vertical and horizontal directions) or the centerof gravity in the entire body, the center of the face, and the like.Alternatively, in a case where the subject is a train or an automobile,a leading end portion in the traveling direction, a driver's seatposition, and the like can be exemplified.

In this example, the position of interest is determined in advanceaccording to the type of the subject, and the image recognitionprocessing unit 5 specifies the attention position of the subjectaccording to the type of the subject.

Note that it is also conceivable to determine the attention position inadvance for each type of the subject according to a user operation.

The image recognition processing unit 5 outputs information related tothe subject (hereinafter referred to as “subject information Is”)detected in the subject detection processing to the control unit 8 inresponse to a request from the control unit 8. Specifically, in a casewhere a subject is detected, the image recognition processing unit 5outputs information indicating a position (subject position Ps asdescribed later) and a range of the subject as the subject informationIs. Furthermore, in a case where the type of the detected subject isrecognized, the image recognition processing unit 5 outputs informationindicating the type together with the information of the position andrange of the subject as the subject information Is.

The recording control unit 6 performs recording and reproduction on arecording medium by, for example, a nonvolatile memory. The recordingcontrol unit 6 performs processing of recording an image file MF such asmoving image data or still image data, a thumbnail image, or the like ona recording medium, for example.

Various actual forms of the recording control unit 6 can be considered.For example, the recording control unit 6 may be configured as a flashmemory built in the imaging device 1 and a write-read circuit thereof,or may be in the form of a card recording-reproducing unit that performsrecording-reproducing access to a recording medium that can be attachedto and detached from the imaging device 1, for example, a memory card(portable flash memory or the like). Furthermore, it may be implementedas a hard disk drive (HDD) or the like as a form built in the imagingdevice 1.

The driver unit 7 is provided with a driver or the like corresponding toeach motor for driving each of the zoom lens, the focus lens, and thediaphragm mechanism in the lens system 2.

These drivers supply a drive current to a corresponding motor inresponse to an instruction from the control unit 8, and cause movementof the focus lens and the zoom lens, opening and closing of thediaphragm blade of the diaphragm mechanism, and the like to be executed.

The control unit 8 includes a microcomputer (arithmetic processing unit)including a central processing unit (CPU), a read only memory (ROM), arandom access memory (RAM), and the like. The RAM described above isused for temporary storage of data, programs, and the like as a workarea at the time of various data processing of the CPU. The ROM is usedfor storing an operating system (OS) for the CPU to control each unit,content files such as image files, application programs for variousoperations, firmware, and the like.

The control unit 8 performs overall control of the imaging device 1 byexecuting a program stored in the above-described ROM or the like. Forexample, the control unit 8 controls operation of each necessary unitwith respect to control of shutter speed of the imaging unit 3,instruction of various signal processing in the camera signal processingunit 4, imaging operation and recording operation according to anoperation of the user using the operation unit 9, reproduction operationof the recorded image file, operation of the lens system 2 such aszooming, focusing, and diaphragm adjustment, user interface operation,and the like.

Furthermore, the control unit 8 executes a program stored in theabove-described ROM or the like to issue an instruction to the platformdevice 50 via the communication unit 11, so as to control the pan/tiltoperation of the platform device 50. That is, the adjustment operationof the imaging direction in the pan and tilt directions is controlled.

Moreover, the control unit 8 of this example controls the platformdevice 50 for tracking the target subject on the basis of detectioninformation of the subject by the image recognition processing unit 5.

Here, “tracking” means that the position of the target subject continuesto coincide with a predetermined target position in the captured image.

Note that a specific example of control for subject tracking performedby the control unit 8 will be described later.

The operation unit 9 collectively indicates input devices for the userto perform various operation inputs. Examples of control elementsincluded in the operation unit 9 can include control elements such asvarious buttons, dials, and/or the like provided in the housing of theimaging device 1. Furthermore, the operation unit 9 is provided with atouch panel 9 a that detects a touch operation on the screen of thedisplay unit 10.

The operation unit 9 detects an operation by the user and outputs asignal according to the detected operation to the control unit 8.

The display unit 10 displays various types of information to the user asthe imaging person. The display unit 10 is, for example, a displaydevice such as a liquid crystal display (LCD) or an organicelectro-luminescence (EL) display arranged in a housing of the imagingdevice 1.

The display unit 10 displays various types of information on the screenon the basis of an instruction from the control unit 8. For example, thedisplay unit 10 displays a reproduced image of the image data read fromthe recording medium in the recording control unit 6.

Furthermore, the image data of the captured image whose resolution hasbeen converted for display by the camera signal processing unit 4 issupplied to the display unit 10, and the display unit 10 displays theimage data of the captured image in response to an instruction from thecontrol unit 8. Thus, what is called a through image (monitoring imageof the subject), which is a captured image during compositionconfirmation, is displayed on the screen.

Furthermore, the display unit 10 displays various operation menus,icons, messages, and the like as a graphical user interface (GUI) on thescreen on the basis of an instruction from the control unit 8.

Hereinafter, the display screen of the display unit 10 is referred to asa “screen 10 a”.

The communication unit 11 performs data communication with an externaldevice according to a predetermined communication format. In particular,the communication unit 11 of this example can perform data communicationwith the communication unit 58 in the platform device 50. Here, the datacommunication between the communication unit 58 and the communicationunit 11 in the platform device 50 may be, for example, wiredcommunication such as USB or wireless communication such as Bluetooth(registered trademark).

1-2. Composition Adjustment

Hereinafter, a composition adjustment method as an embodiment will bedescribed with reference to FIGS. 4 to 12.

First, in the present embodiment, a target composition is designated byan operation on the screen 10 a that displays a captured image of theimaging device 1 whose composition is adjusted by the platform device50. Specifically, the imaging device 1 of the present embodimentreceives an operation of designating a position on the screen 10 a thatdisplays a captured image as a designation operation of a targetcomposition, and controls the platform device 50 to adjust thecomposition to the target composition. More specifically, in thisexample, the operation of designating the position on the screen 10 a asdescribed above is received as an operation of designating acomposition. That is, in this case, designation operation information ofthe composition (composition designation operation information) acquiredfrom the operation of designating the composition includes informationof the designated position on the screen 10 a. Then, in the compositionadjustment method of this example, the composition of the imaging deviceis adjusted on the basis of the subject information corresponding to thedesignated position by such composition designation operationinformation. The subject information is information regarding thesubject in the captured image, and includes at least informationindicating presence or absence of the detected subject. By performingcomposition adjustment based on such subject information, it is possibleto perform composition adjustment according to at least the presence orabsence of the detected subject.

Here, in this example, the target composition means a target compositionwhen tracking the subject. That is, the composition here includes atleast an element with respect to which position in the captured imagethe subject to be tracked is to be arranged at.

A specific composition adjustment method will be described withreference to FIGS. 4 and 5.

As illustrated in FIG. 4, it is assumed that a captured image in whichtwo detected subjects as a subject S1 and a subject S2 are captured isdisplayed on the screen 10 a of the display unit 10. Note that, at thistime, it is assumed that tracking of the detected subject has not beenperformed yet.

In this example, in a state where the detected subject is displayed onthe screen 10 a as described above, tracking of the touched detectedsubject is started in response to an operation of touching the detectedsubject (an operation of touching the subject S2 in the example of FIG.4). Specifically, the processing of controlling the panning-tiltingoperation of the platform device 50 is started such that the position ofthe touched detected subject is maintained at the position of thetouched detected subject at the time of touch.

Here, a state where the detected subject is designated by the touchoperation is, in other words, a state where the subject informationcorresponding to the designated position by the touch operationindicates the detected subject.

Here, subject tracking control processing in this example will bedescribed with reference to FIG. 6.

In this example, proportional-integral-differential (PID) control isemployed for control of the platform device 50 for tracking the subject.Specifically, as illustrated in FIG. 6A, an error d (position error d)between a position Ps of the subject as the tracking target and a targetposition Pt of tracking is calculated, and a control value C of theplatform device 50 is calculated on the basis of the error d by

C=P(d)+I(d)+D(d).

Note that P(d), I(d), and D(d) mean a proportional term, an integralterm, and a differential term, respectively, for the error d.

FIG. 6B is a functional block diagram illustrating a functionalconfiguration of the control unit 8 for achieving the subject trackingcontrol processing. Note that since the control unit 8 controls the pandirection and the tilt direction as control for subject tracking, theconfiguration of the control system illustrated in FIG. 6B is actuallyprovided for each of the pan direction and the tilt direction.

As illustrated, the control unit 8 includes a position error calculationprocessing unit F1, a control value calculation processing unit F2, anda drive amount conversion unit F3 as functional units for achievingsubject tracking. At a time of tracking the subject, the position errorcalculation processing unit F1 receives input of the subject positionPs, which is the position of the subject as the tracking target, and thetarget position Pt of tracking, and calculates the error d thereof.

Here, in a case where a plurality of subjects is detected in thecaptured image as illustrated in FIG. 4, as the information of thesubject position Ps, information of the position of the subjectdesignated by the touch operation among information of positions of aplurality of subjects input from the image recognition processing unit 5is used. Furthermore, the target position Pt is set as information ofthe position of the subject designated by the touch operation when thetouch operation of the subject is performed.

Based on the error d input from the position error calculationprocessing unit F1, the control value calculation processing unit F2calculates the control value C by performing calculation according tothe above-described equation of PID control.

The drive amount conversion processing unit F3 converts the controlvalue C calculated by the control value calculation processing unit F2,that is, a value in units of the number of pixels of the captured imageinto a value of an angle in a rotation direction as the pan direction orthe tilt direction. The conversion can be performed on the basis ofimaging parameters (focal length, size information of the imagingelement, and the like) of the imaging device 1.

The control unit 8 outputs the value obtained by the conversionprocessing of drive amount conversion processing unit F3, that is, thevalue representing the driving amount in the pan direction or the tiltdirection to the platform device 50 via the communication unit 11.

In the platform device 50, the value output from the control unit 8 viathe communication unit 11 as described above is input to the drivecontrol unit 57 via the communication unit 58. The drive control unit 57drives the actuator 56 by a drive amount corresponding to the inputvalue. Thus, the direction of the imaging device 1 is adjusted so thatthe position (Ps) of the subject to be tracked coincides with the targetposition (Pt) in the captured image, and the tracking of the subject isachieved.

Here, as described above, in this example, in a case where the operationof touching the detected subject in the screen 10 a is performed, theposition of the detected subject at the time of touch is set as thetarget position Pt in the tracking control. That is, the operation oftouching the detected subject functions not only as the operation ofdesignating the subject to be tracked but also as the operation ofdesignating the arrangement position of the subject to be tracked in thescreen.

Since the designation of the subject to be tracked and the designationof the arrangement position of the subject to be tracked in the screencan be achieved by a single operation of touching the detected subject,it is possible to reduce an operation burden on the user related tocomposition adjustment, and improve the operability.

The description is returned to FIG. 4.

It is assumed that after the detected subject in the screen 10 a istouched, the finger that has made the touch is moved on the screen 10 aas indicated by a thick arrow in FIG. 4 while maintaining the touchstate.

In this example, the operation of tracing the screen 10 a with thefinger touching the subject while remaining in the touched state asdescribed above is received as a movement instruction operation of thearrangement position (that is, the arrangement position in the capturedimage) of the touched subject (that is, the tracking target subject) inthe screen 10 a.

FIG. 5 illustrates a state of the screen 10 a when the arrangementposition of the subject is changed according to the tracing operationindicated by the thick arrow in FIG. 4.

The control unit 8 controls the platform device 50 such that the touchedsubject is moved by an amount corresponding to the amount of tracing inthe direction traced by the tracing operation according to the tracingoperation as described above.

At this time, the tracking state of the touched subject is maintained.Specifically, in the process in which the touch position graduallychanges by the tracing operation, the control unit 8 sequentiallyupdates the target position Pt to the touch position at that time everytime the touch position changes. Thus, the position of the touchedsubject in the screen 10 a can be smoothly changed in conjunction withthe tracing operation.

Here, in this example, in response to the tracing operation after thedetected subject is touched as described above, the arrangement positionof the touched subject in the screen 10 a is moved in the tracingdirection by the tracing operation, and for this purpose, the controlunit 8 controls the platform device 50 such that the imaging directionof the imaging device 1 changes in the direction opposite to the tracingdirection.

Note that the tracing operation as described above is, in other words,an operation including designation of a direction in the plane of thescreen 10 a. Furthermore, such a tracing operation constitutes a part ofthe composition designation operation similarly to the operation oftouching the detected subject (the operation of designating a positionon the screen 10 a).

FIGS. 7 and 8 are diagrams for describing behavior in a case where thetracing operation as described above is performed after a subject otherthan the detected subject is touched.

It is assumed that, as illustrated in FIG. 7, after a subject as abackground portion other than the subjects S1 and S2, which are detectedsubjects, is touched on the screen 10 a, a tracing operation with thefinger touching is performed.

In the present embodiment, the tracing operation after touching thesubject other than the detected subjects in this manner is received asan operation of directly giving an instruction on the direction ofchange in the imaging direction. That is, for example, in a case wherethe tracing operation in the right direction is performed as illustratedin FIG. 7, the imaging direction is changed by an amount correspondingto the amount traced in the right direction as illustrated in FIG. 8.That is, the control unit 8 controls the platform device 50 such thatthe imaging direction is changed by the amount corresponding to thetracing amount in the tracing direction in response to the tracingoperation after touching the subject other than the detected subjects.

Here, in a case where a subject other than the detected subjects istouched, the control unit 8 does not perform tracking control of thesubject.

As described above, in this example, even in the same tracing operation,the changing direction of the imaging direction is reversed between acase of touching the detected subject and a case of touching the subjectother than the detected subjects.

Here, the state where the subject other than the detection subjects isdesignated by the touch operation is, in other words, a state where thesubject information corresponding to the designated position by thetouch operation does not indicate the detected subject.

Furthermore, in the present embodiment, an angle of view can be changedby an operation on the screen.

FIGS. 9 and 10 are diagrams for explaining an operation of changing anangle of view.

In this example, after the first finger touches the screen 10 a, in acase where another finger (second finger) touches the screen 10 a whilethe touch state of the first finger is continued, the angle of view ischanged according to the operation of tracing the screen 10 a with thesecond finger.

Specifically, in this example, in a case where the touch position of thesecond finger changes in a direction away from the touch position of thefirst finger, the control unit 8 controls the driver unit 7 so that thezoom lens is driven in a direction of decreasing the angle of view (thatis, zooming in). Thus, a change in the angle of view as illustrated as atransition from FIG. 9 to FIG. 10 is achieved.

On the other hand, in a case where the touch position of the secondfinger changes in a direction to approach the touch position of thefirst finger, the control unit 8 controls the driver unit 7 so that thezoom lens is driven in a direction of increasing the angle of view (thatis, zooming out). The change in the angle of view in this case is achange represented as a transition from FIG. 10 to FIG. 9.

Note that in FIGS. 9 and 10, the change of the angle of viewcorresponding to the case where the detected subject is touched with thefirst finger has been exemplified, but also in a case where a subjectother than the detected subjects is touched with the first finger, thecontrol unit 8 performs control of the angle of view similar to thatdescribed above.

The tracing operation with the second finger described above is, inother words, an operation including designation of a direction in theplane of the screen 10 a.

Furthermore, the change (control) of the angle of view is one aspect ofthe change (control) of the imaging range by the imaging device 1.

Furthermore, in the present embodiment, in a case where the operation oftouching the screen 10 a is performed again after the subject to betracked is designated by the touch operation, in other words, in a casewhere an operation of double-tapping the screen 10 a including the tapof the detected subject is performed, adjustment to a composition inwhich the tracking target subject is arranged at a predeterminedposition in the captured image is performed.

Note that the “tap” mentioned here means a series of operations fromtouching the screen 10 a with a finger to releasing the finger from thescreen 10 a. Furthermore, the double tap operation here means anoperation of performing a second tap within a predetermined time afterthe first tap.

FIGS. 11 and 12 are explanatory diagrams of composition adjustmentaccording to such a double tap operation.

In this example, in a case where the double tap operation as describedabove is performed, adjustment is performed to a composition in whichthe tracking target subject (the subject S2 in the example of FIG. 11)designated by the first tap is arranged at a center position Pc of theangle of view (the center position of the captured image) (see FIG. 12).

Specifically, in response to the double tap operation as describedabove, the control unit 8 updates the target position Pt of tracking tothe center position Pc and continues the tracking control of thesubject.

1-3. Processing Procedure

An example of a specific processing procedure for achieving thecomposition adjustment method as the embodiment described above will bedescribed with reference to a flowchart of FIG. 13.

Note that the processing illustrated in FIG. 13 is executed by thecontrol unit 8 according to a program stored in a memory such as a ROM.

In FIG. 13, in step S101, the control unit 8 waits until there is atouch on the screen 10 a, and in a case where there is a touch on thescreen 10 a, the control unit 8 executes processing of acquiring thesubject information Is from the image recognition processing unit 5 assubject information acquisition processing in step S102, and determineswhether or not it is a touch on the detected subject in step S103. Thatis, it is determined whether or not the position on the screen 10 adesignated by the touch operation detected in step S101 (that is, thetouch start position on the screen 10 a) is a position within the rangeof the subject included in the subject information Is acquired in stepS102.

In step S103, in a case where a negative result is obtained indicatingthat the position on the screen 10 a designated by the touch operationis not the position within the range of the subject included in thesubject information Is and is not the touch of the detected subject, thecontrol unit 8 proceeds to step S104 to turn OFF a tracking executionflag, and executes normal drive mode processing in subsequent step S105.

Here, the tracking execution flag is a flag indicating whether or not toexecute the subject tracking control processing described with referenceto FIG. 6, and in a case where the tracking execution flag is turned ON,the control unit 8 starts the subject tracking control processing inparallel with the processing illustrated in FIG. 13. When the trackingexecution flag is OFF, the control unit 8 does not execute the trackingcontrol processing.

Note that the initial value of the tracking execution flag is OFF, butthe reason why the tracking execution flag is turned OFF in step S104 isto stop the tracking control of the subject when an operation to touch asubject other than the detected subject is performed after the detectedsubject is touched and the tracking control of the subject is started(step S103: N).

Note that the normal drive mode processing in step S105 is a process forchanging the imaging direction according to the tracing operation by thefinger touching the subject other than the detected subject, or foradjusting the angle of view according to the tracing operation by thesecond finger, the details of which will be described later.

Furthermore, in a case where an affirmative result indicating that thetouch is the touch of the detected subject is obtained in step S103, thecontrol unit 8 proceeds to step S106 to turn ON the tracking executionflag, and then executes subject tracking mode processing in step S107.

Here, in response to the turning ON of the tracking execution flag instep S106, the control unit 8 starts the subject tracking controlprocessing based on the setting information (setting information of thetracking target subject and the target position Pt) by the processing insteps S301 and S302 described in FIG. 15. In the tracking controlprocessing, the control unit 8 executes acquisition of the subjectinformation Is (in particular, information of the subject position Ps ofthe tracking target subject) from the image recognition processing unit5 for each frame, and uses the subject information Is for calculation ofthe error d and the like.

Note that the normal drive mode processing in step S105 is processingfor performing control to move the subject position in the screen 10 aaccording to the tracing operation by the finger touching the detectedsubject, or for adjusting the angle of view according to the tracingoperation by the second finger, the details of which will be describedlater.

The control unit 8 advances the processing to step S108 in each of acase where the normal drive mode processing of step S105 is executed anda case where the subject tracking mode processing of step S107 isexecuted.

In step S108, the control unit 8 determines whether or not processingend condition is satisfied. The processing end condition here means anend condition of the processing illustrated in FIG. 13, and isconceivably set to a condition that, for example, there is performed anoperation of switching an operation mode of the imaging device 1 from animaging mode in which a captured image as a through image is displayedon the screen 10 a to a reproduction mode in which an image reproducedfrom a recording medium by the recording control unit 6 is displayed onthe screen 10 a, or the like.

In a case where it is determined that the processing end condition isnot satisfied, the control unit 8 returns to step S101 and waits againfor a touch operation on the screen 10 a.

Thus, in a case where the screen 10 a is touched again after thedetected subject or the subject other than the detected subject istouched and the touched finger is separated from the screen 10 a, thenormal drive mode processing in step S105 or the subject tracking modeprocessing in step S107 is executed depending on whether the touch is atouch on the subject other than the detected subject or a touch on thedetected subject.

On the other hand, in a case where it is determined in step S108 thatthe processing end condition is satisfied, the control unit 8 determineswhether or not the tracking execution flag is ON in step S109, and thenif the tracking execution flag is ON, the control unit 8 sets thetracking execution flag to OFF in step S110 and ends the series ofprocessing illustrated in FIG. 13, and if the tracking execution flag isnot ON, the control unit 8 passes the processing of step S110 and endsthe series of processing illustrated in FIG. 13.

FIG. 14 is a flowchart of the normal drive mode processing in step S105.

The processing illustrated in FIG. 14 is processing repeatedly executedfor each frame of the captured image by the standby processing in stepS211 until it is detected in step S210 that the first finger (that is,the finger for which the touch operation is detected in step S101) hasmoved away from the screen 10 a.

In FIG. 14, in step S201, the control unit 8 determines whether or not adifferent finger touch flag is ON. The different finger touch flag is aflag indicating whether or not the second finger (see the operation ofchanging the angle of view described above) is in a touched state, andis turned ON in step S205 as described later in a case where there is atouch of the second finger, and is turned OFF in step S209 in responseto moving away of the second finger from the screen 10 a. At a time ofstart of the processing illustrated in FIG. 14, the different fingertouch flag is OFF.

If the different finger touch flag is not ON in step S201, the controlunit 8 proceeds to step S202 to determine whether or not there is atouch of the second finger, and then if there is no touch of the secondfinger, the control unit 8 proceeds to step S203 to determine whether ornot the first finger has moved.

In a case where the first finger has moved, the control unit 8 proceedsto step S204 and executes processing of outputting control valuesaccording to the movement. That is, processing is executed foroutputting control values (control values of both the actuator in thepan direction and the actuator in the tilt direction) of the platformdevice 50 according to the moving direction and the moving amount of thefirst finger to the platform device 50 via the communication unit 11.

Thus, in a case where the subject other than the detected subject istouched and the touched finger is traced in any direction on the screen10 a, the pan and tilt operations of the platform device 50 arecontrolled such that the imaging direction of the imaging device 1 ischanged by an amount corresponding to the tracing amount in the tracingdirection.

In response to the execution of the output processing of step S204, thecontrol unit 8 executes one-frame standby processing of step S211 andreturns to step S201. Thus, while the movement of the first finger iscontinuously performed, the processing of step S204 is repeated for eachframe, so that the imaging direction is continuously changed.

Furthermore, in a case where it is determined in step S203 that thefirst finger has not moved, the control unit 8 proceeds to step S210 anddetermines whether or not the first finger has moved away from thescreen 10 a.

If the first finger is not moved away from the screen 10 a, the controlunit 8 proceeds to the one-frame standby processing in step S211. Thatis, while the first finger maintains the touch state, the processingillustrated in FIG. 14 is repeated for each frame.

On the other hand, in a case where the first finger has moved away fromthe screen 10 a, the control unit 8 ends the normal drive modeprocessing in step S105.

Furthermore, in a case where it is determined in step S202 that thesecond finger touch has been made, the control unit 8 turns ON thedifferent finger touch flag in step S205, executes the one-frame standbyprocessing in step S211, and then returns to step S201.

In a case where it is determined in step S201 that the different fingertouch flag is ON, the control unit 8 proceeds to step S206 anddetermines whether or not the second finger has moved. If the secondfinger has moved, the control unit 8 proceeds to step S207 and performszooming control according to the movement. That is, if the movement ofthe second finger is in a direction away from the touch position of thefirst finger, the driver unit 7 is controlled to drive the zoom lens ina zoom-in direction, and if the movement of the second finger is in adirection approaching the touch position of the first finger, the driverunit 7 is controlled to drive the zoom lens in a zoom-out direction.

In response to the execution of the zooming control in step S207, thecontrol unit 8 executes the one-frame standby processing in step S211,and returns to step S201. Thus, while the movement of the second fingeris continued, the zooming control in step S207 is repeated to continuethe change of the angle of view.

On the other hand, in a case where it is determined in step S206 thatthe second finger has not moved, the control unit 8 proceeds to stepS208 and determines whether or not the second finger has moved away fromthe screen 10 a. In a case where it is determined that the second fingerhas not moved away from the screen 10 a, the control unit 8 proceeds tostep S210 described above and determines whether or not the first fingerhas moved away from the screen 10 a. That is, in this example, even ifthe second finger has not moved away from the screen 10 a, in a casewhere the first finger has moved away from the screen 10 a, the normaldrive mode processing in step S105 ends. That is, the user releases thefirst finger touching the subject other than the detected subject fromthe screen 10 a, and then touches the detected subject, by which it ispossible to given an instruction on tracking of the touched detectedsubject or changing of the arrangement position of the tracking targetsubject by the tracing operation.

On the other hand, in a case where it is determined that the secondfinger has moved away from the screen 10 a, the control unit 8 proceedsto step S209 to turn OFF the different finger touch flag, executes theone-frame standby processing of step S211, and then returns to stepS201.

Thus, even after the second finger has moved away from the screen 10 a,while the touch state of the first finger continues, control to changethe imaging direction according to the tracing operation of the firstfinger is performed.

FIG. 15 is a flowchart of the subject tracking mode processing in stepS107.

Note that in the following description, the same step numbers will beused for processing similar to the already described processing, anddescription thereof is omitted.

In the subject tracking mode processing in step S107, the control unit 8first sets the tracking target subject in step S301. That is, thedetected subject designated by the touch operation detected in step S101is set as the tracking target subject.

In subsequent step S302, the control unit 8 sets the target position Pt.That is, the position (subject position Ps) of the detected subjectdesignated by the touch operation detected in step S101 is set as thetarget position Pt of tracking.

Next, in step S303, the control unit 8 determines whether or not aretouch standby flag is ON. The retouch standby flag is a flag foridentifying whether or not it is a standby state for a second tapoperation in the double tap operation described above, and ON representsthe standby state, and an initial value is OFF. As will be describedlater, the retouch standby flag is turned ON by the processing of stepS305 in response to the moving away of the first finger from the screen10 a (step S210: Y).

In step S303, when the retouch standby flag is not ON, the control unit8 advances the processing to step S201. Here, processing from step S201to step S211 is similar to the processing described with reference toFIG. 14, and thus redundant description is avoided.

However, in this case, in a case where it is determined in step S203that the first finger has moved, the control unit 8 proceeds to stepS304 and executes update processing of the target position Pt accordingto the movement. That is, the target position Pt is updated to thecurrent touch position of the first finger.

In response to the execution of the update processing in step S304, thecontrol unit 8 proceeds to the one-frame standby processing in stepS211. Thus, while the first finger is moving, the update processing instep S304 is repeatedly executed for each frame. Therefore, the movementof the tracking target subject on the screen 10 a becomes smooth.

Furthermore, in the processing of FIG. 15, in a case where it isdetermined in step S210 that the first finger has moved away from thescreen 10 a, the control unit 8 turns ON the retouch standby flag instep S305, performs the one-frame standby processing in step S211, andthen returns to step S303.

In a case where it is determined in step S303 that the retouch standbyflag is ON, the control unit 8 advances the processing to step S306. Theprocessing in and after step S306 is processing corresponding to thedouble tap operation.

First, in step S306, the control unit 8 determines whether or not thereis a retouch on the screen 10 a, and if there is no retouch on thescreen, the control unit 8 determines in step S307 whether or not aretouch waiting time is equal to or longer than a predetermined time.The retouch waiting time means an elapsed time from when it isdetermined in step S210 that the first finger has moved away.

If the retouch waiting time is not equal to or longer than thepredetermined time, the control unit 8 returns to step S306. That is, bythe processing of steps S306 and S307, the control unit 8 waits for thetouch operation (that is, the touch operation as a second tap) within apredetermined time after the first finger is released.

In a case where it is determined in step S307 that there is a retouch,the control unit 8 proceeds to step S308 and performs processing ofupdating the target position Pt to the screen center position (thecenter position Pc of the angle of view). Thus, in response to thedouble tap operation being performed, the composition is adjusted suchthat the position (Ps) of the tracking target subject designated by thefirst tap coincides with the center position Pc.

In response to the execution of the update processing in step S308, thecontrol unit 8 executes flag OFF processing in step S309, that is, theprocessing of turning OFF the different finger touch flag and theretouch standby flag, and terminates the series of processingillustrated in FIG. 15.

As can be seen with reference to FIG. 13, after executing the subjecttracking mode processing in step S107, if it is not determined in stepS108 that the processing end condition is satisfied, the control unit 8proceeds to the processing in step S101. Thus, in this example, evenafter the tracking target subject is moved to the center of the screen10 a by the double tap operation, the touch operation on the screen 10 ais received, the processing of FIG. 14 is executed if the touchoperation is an operation of touching the subject other than thedetected subject, and the processing of FIG. 15 is executed if the touchoperation is an operation of touching the detected subject.

2. Second Embodiment

Next, a second embodiment will be described.

In the second embodiment, a parameter corresponding to a type of asubject is used in the subject tracking control processing.

Note that in the second embodiment, configurations of the imaging device1 and the platform device 50 are similar to those of the case of thefirst embodiment, and thus redundant description is avoided.

FIG. 16 is a flowchart illustrating an example of a specific processingprocedure for achieving a composition adjustment method as the secondembodiment.

In FIG. 16, the processing of steps S101 to S110 is similar to those inthe case of FIG. 13.

In this case, in a case where it is determined in step S103 that thetouch is a touch of the detected subject, the control unit 8 proceeds tostep S401, executes the parameter setting processing according to thesubject type, and then proceeds to step S106.

FIG. 17 is a flowchart illustrating parameter setting processingaccording to a subject type in step S401.

First, in step S410, the control unit 8 checks the type of the touchedsubject. Here, the image recognition processing unit 5 in this examplerecognizes three types of first type, second type, and third type as thetype of the detected subject. In step S410, the control unit 8 checkswhether the type of the touched subject is the first type, the secondtype, or the third type.

Here, in this example, the first type is “bird” (see FIG. 18A), thesecond type is “person” (see FIG. 18B), and the third type is “dog” (seeFIG. 18C).

Note that, regarding the first type, the “bird” is merely an example,and the first type is only required to be a type of a subject that tendsto be imaged on a telephoto side and moves relatively fast verticallyand horizontally like the “bird”.

Similarly, regarding the second type, the “person” is an example, and isonly required to be a type of a subject that relatively slowly movesvertically and horizontally, and for the third type, the “dog” is anexample, and is only required to be a type of a subject that relativelyslowly moves vertically but relatively quickly moves horizontally.

In a case where the type of the touched subject is the first type, thecontrol unit 8 proceeds to step S411, sets the first type parameter, andends the processing of step S401. Furthermore, in a case where the typeof the touched subject is the second type, the control unit 8 sets thesecond type parameter in step S412 and ends the processing of step S401,and in a case where the type of the touched subject is the third type,the control unit 8 sets the third type parameter in step S413 and endsthe processing of step S401.

Here, in this example, specific examples of the parameter used in thetracking control processing can include a control cycle of trackingcontrol and a gain (gain of proportional term, integral term, andderivative term) in PID control. As the processing of setting the firstparameter (parameter in the case of “bird”) in step S411, the controlunit 8 sets a parameter for increasing responsiveness of tracking foreach of the tracking control processing in the pan direction and thetracking control processing in the tilt direction. Specifically, forexample, the control cycle is set to the shortest cycle among settablecycles, and the gain in the PID control is set to the highest valueamong settable values.

Furthermore, as the processing of setting the second parameter(parameter in the case of “person”) in step S412, the control unit 8sets a parameter for decreasing the responsiveness of tracking for eachof the tracking control processing in the pan direction and the trackingcontrol processing in the tilt direction. Specifically, for example, thecontrol cycle is set to a longer cycle (for example, a double cycle)than the above-described shortest cycle, and the gain in the PID controlis set to a value lower than the above-described maximum value.

Moreover, as the processing of setting the third parameter (parameter inthe case of “dog”) in step S413, the control unit 8 sets a parameterthat increases the responsiveness of tracking with respect to thetracking control processing in the pan direction and decreases theresponsiveness of tracking with respect to the tracking controlprocessing in the tilt direction.

Here, in this example, in a case where the control cycle of the trackingcontrol is set to a cycle that is twice the shortest cycle as in thesecond parameter described above, in the tracking control processing,the control unit 8 integrates the tracking result (control value C)calculated in the shortest cycle, in other words, the tracking resultsof two times, and uses the integrated value for control of the platformdevice 50. Thus, the influence of noise can be reduced, and stability oftracking control can be improved.

By performing the parameter setting according to the type of the subjectas described above, it is possible to set an optimum parameter accordingto movement characteristics for each type of the subject regarding thetracking control of the subject, and it is possible to improve theaccuracy of the tracking control.

Furthermore, in this example, the parameter setting is individuallyperformed for each of the tracking control in the pan direction and thetracking control in the tilt direction according to the type of thesubject, but this configuration enables tracking with excessivelyincreased responsiveness to be prevented from being performed in adirection in which it is unnecessary to increase the trackingresponsiveness, and the stability of the tracking control can beimproved.

Here, in the above description, the parameter setting example in whichthe tracking responsiveness in the pan direction is increased and thetracking responsiveness in the tilt direction is decreased according tothe type of the “dog” has been described, but depending on the type ofthe subject, the parameter setting can also be performed in which thetracking responsiveness in the pan direction is decreased and thetracking responsiveness in the tilt direction is increased conversely.

For example, when the type of the subject is a relatively largeimmovable body such as a house or a building, and the imaging personperforms imaging while moving around the subject, it is conceivable toperform parameter setting for decreasing the tracking responsiveness inthe pan direction and increasing the tracking responsiveness in the tiltdirection. Alternatively, it is also conceivable to similarly decreasethe tracking responsiveness in the pan direction and increase thetracking responsiveness in the tilt direction corresponding to a type ofsubject assumed to move violently only in the vertical direction by atrampoline or the like.

Furthermore, in the second embodiment, the control unit 8 starts theprocessing for parameter adjustment illustrated in FIG. 19 in responseto the execution of the parameter setting processing illustrated in FIG.17.

Here, the processing of FIG. 19 is executed for each of the trackingcontrol processing in the pan direction and the tracking controlprocessing in the tilt direction.

In FIG. 19, in step S420, the control unit 8 determines whether or notthe current setting parameter is a high responsiveness parameter. Thehigh responsiveness parameter means a parameter that increases thetracking responsiveness described in the processing of steps S411 toS412 described above. For example, the parameters set in the pandirection and the tilt direction in step S411 are high responsivenessparameters.

In a case where it is determined that the current setting parameter isthe high responsiveness parameter, the control unit 8 proceeds to stepS421 and determines whether or not the position error d is smaller thana threshold value TH1. In a case where the position error d is smallerthan the threshold value TH1, the control unit 8 performs processing ofchanging to the low responsiveness parameter in step S422. The lowresponsiveness parameter means a parameter that decreases the trackingresponsiveness described in the processing of steps S411 to S412described above. In step S422, the parameter used in the trackingcontrol processing is changed to the low responsiveness parameter.

Thus, in a case where the position error d is extremely small after thesetting of the high responsiveness parameter, the parameter of thetracking control can be adjusted to the low responsiveness parameter.

In response to the execution of the change processing in step S422, thecontrol unit 8 advances the processing to step S425. Furthermore, in acase where it is determined in step S421 that the position error d isnot smaller than the threshold value TH1, the control unit 8 proceedsthe processing to step S425 without executing the change processing ofstep S422.

Furthermore, in a case where it is determined that the current settingparameter is not the high responsiveness parameter in step S420, thecontrol unit 8 proceeds to step S423 and determines whether or not theposition error d is larger than a threshold value TH2. In a case wherethe position error d is larger than the threshold value TH2, the controlunit 8 performs processing of changing to the high responsivenessparameter in step S424.

Thus, in a case where the position error d is extremely large after thesetting of the low responsiveness parameter, the parameter of thetracking control can be adjusted to the high responsiveness parameter.

In response to the execution of the change processing in step S424, thecontrol unit 8 advances the processing to step S425. Furthermore, in acase where it is determined in step S423 that the position error d isnot larger than the threshold value TH1, the control unit 8 proceeds theprocessing to step S425 without executing the change processing of stepS424.

In step S425, the control unit 8 determines whether or not the trackingexecution flag is OFF, and if the tracking execution flag is not OFF,the control unit 8 returns to step S420, or if it is OFF, the controlunit 8 terminates the series of processing illustrated in FIG. 19. Thatis, the processing illustrated in FIG. 19 is executed during a period inwhich the subject tracking control processing is being executed.

By performing the parameter adjustment based on the position error d asdescribed above, it is possible to achieve the parameter adjustmentbased on the determination result as to whether or not the set parameteris suitable for the actual movement of the subject, and the stability ofthe tracking control can be improved.

Note that FIG. 19 is an example in which the parameter adjustment isperformed on the basis of the comparison result between the positionerror d and the threshold value TH, but alternatively, the parameteradjustment can be performed on the basis of the comparison resultbetween the control value C and the corresponding threshold value. Evenin a case of such parameter adjustment, there is no difference in thatthe parameter adjustment is performed on the basis of the error betweenthe position of the subject and the tracking target position.

In the above description, an example of two-stage adjustment of the highresponsiveness parameter and the low responsiveness parameter has beendescribed, but the parameter adjustment can be adjustment between threeor more stages of parameters.

3. Third Embodiment 3-1. First Example

In a third embodiment, composition adjustment is performed on the basisof simplified model information of a human body.

Note that also in the third embodiment, configurations of the imagingdevice 1 and the platform device 50 are similar to those of the case ofthe first embodiment, and thus redundant description is avoided.

First, the simplified model information of the human body will bedescribed with reference to FIG. 20.

As illustrated in FIG. 20, the simplified model information of the humanbody is information indicating an arrangement state of a plurality ofparts constituting the human body such as eyes, ears, a nose, a mouth, aneck, shoulders, elbows, wrists, buttocks, knees, and ankles in thehuman body. In the third embodiment, in a case where the detectedsubject is a subject as a human, the image recognition processing unit 5generates such simplified model information of the human body for thesubject.

Specifically, the image recognition processing unit 5 generatesinformation of detection points (black circles in the drawings) of aspecific part such as an eye, a neck, and a shoulder in a detectedsubject as a person and lines connecting the detection points as thesimplified model information. At this time, each detection point isconnected to a specific other detection point by a line. The detectionpoints connected by lines are defined, thereby forming a simple modelexpressing the human body by points and lines.

In this case, the image recognition processing unit 5 outputs thesimplified model information to the control unit 8 as a part of thesubject information Is in response to a request from the control unit 8.Also in this example, the control unit 8 executes acquisition of thesubject information Is from the image recognition processing unit 5 foreach frame during the subject tracking.

Hereinafter, a first example and a second example will be described asexamples of composition adjustment based on the simplified modelinformation as described above.

In a first example, composition switching is performed between aplurality of compositions in which combinations of regions of a subjectappearing in a captured image are different on the basis of thesimplified model information of the human body.

Specifically, composition switching is performed among a composition K1,a composition K2, and a composition K3 illustrated in FIG. 21. In FIG.21, the composition K1 is a composition as a “full shot” (whole-bodycomposition) in which the combination of the parts of the subjectappearing in the captured image is a combination of the “eyes, ears, anose, a mouth, a neck, shoulders, elbows, wrists, buttocks, knees, andankles”. Furthermore, the composition K2 is a composition as a “kneeshot” (composition of the knees and above) in which the combination ofthe parts of the subject shown in the captured image is a combination of“eyes, ears, nose, mouth, neck, shoulder, elbows, wrists, buttocks, andknees” in which only “ankles” are removed from the composition K1, andthe composition K3 is a composition as a “bust shot” (composition of thehalf body) in which the combination of the parts of the subject shown inthe captured image is a combination of the “eyes, ears, a nose, a mouth,a neck, shoulders, elbows” in which the “knees, buttocks, and wrists”are further removed from the composition K2.

FIG. 22 illustrates an example of the captured image corresponding toeach of the compositions K1, K2, and K3, in which FIG. 22A correspondsto the composition K1 (full shot), FIG. 22B corresponds to thecomposition K2 (knee shot), and FIG. 22C corresponds to the compositionK3 (bust shot).

In the first example, these compositions K1, K2, and K3 are switchedaccording to a touch operation on the screen 10 a. Specifically, in astate where the detected subject as a person is displayed on the screen10 a, the control unit 8 in this case switches the compositions K1, K2,and K3 according to the tap operation on the detected subject on thescreen 10 a after the detected subject is arranged at the center of thescreen 10 a by the above-described double tap operation. Morespecifically, the control unit 8 in this case switches the compositionin a toggling manner as composition K1→composition K2→compositionK3→composition K1→ . . . every time the tap operation is performed onthe detected subject after the double tap operation.

Note that the switching of the composition K is not limited to theexample performed according to the tap operation after the double tapoperation as described above, and can be performed according to, forexample, a zoom operation or the like after the tracking target subjectis designated by the touch operation of the first finger.

FIG. 23 is a flowchart illustrating an example of a specific processingprocedure for achieving the composition adjustment method as the firstexample as described above.

Note that the control unit 8 in this case also executes the processingillustrated in FIG. 13 as in the case of the first embodiment. However,the control unit 8 in this case executes the processing illustrated inFIG. 23 instead of the processing illustrated in FIG. 15 as the subjecttracking mode processing in step S107.

The processing illustrated in FIG. 23 is different from the processingillustrated in FIG. 15 in that composition switching processing based onthe simplified model information in step S501 is executed after theprocessing in step S309 is executed.

FIG. 24 is a flowchart illustrating the composition switching processingbased on the simplified model information in step S501.

First, in step S510, the control unit 8 resets a tap numberidentification value c to zero. The tap number identification value c isa value for identifying the number of taps after the double tapoperation, and every time the re-tapping on the tapped subject isdetected in step S511, the value is updated in a toggling manner asc=0→1→2→0→ . . . by the processing in steps S516 and S521.

In step S511 following step S510, the control unit 8 determines whetheror not the tapped subject has been re-tapped. The tapped subject meansthe detected subject designated by the touch operation detected in stepS101. In step S511, it is determined whether or not an operation oftapping within the range of the tapped subject has been performed.

Describing for confirmation, at the time when the composition switchingprocessing in step S501 is started, a touch for positioning the detectedsubject at the center of the screen has already been performed (seesteps S210, S305 to S308 in FIG. 23). In step S511, the expression“re-tap” is used to determine the presence or absence of a tap operationon the detected subject from this state.

In a case where it is determined that the tapped subject has not beenre-tapped, the control unit 8 proceeds to step S512 and determineswhether or not an area other than the tapped subject has been touched,and in a case where it is determined that an area other than the touchedsubject has not been touched, the control unit 8 returns to step S511.That is, the control unit 8 waits for either a re-tap of the tappedsubject or a touch of an area other than the touched subject by theprocessing of steps S511 and S512.

In a case where it is determined in step S512 that there is a touch ofan area other than the tapped subject, the control unit 8 returns tostep S102. Thus, if the touch of the area other than the tapped subjectperformed after the double tap operation is on the detected subjectother than the tapped subject, the processing of step S107 is executed(that is, the tracking of the detected subject other than the tappedsubject is switched, enabling adjustment and the like of the subjectarrangement position according to the tracing operation). Alternatively,if the touch of the area other than the tapped subject performed afterthe double tap operation is a touch on the area other than the detectedsubject, the processing of step S105 is executed (that is, the subjecttracking is released, enabling adjustment and the like of the imagingdirection according to the tracing operation).

On the other hand, in a case where it is determined in step S511 thatthe tapped subject is re-tapped, the control unit 8 proceeds to stepS513 and checks the tap number identification value c. If the tap numberidentification value c is zero, the control unit 8 proceeds to step S514and calculates the focal length corresponding to the full shot(composition K1) as a target focal length. In other words, the targetfocal length is calculated so as to achieve a composition in which thecombination of portions appearing in the captured image is a combinationof “eyes, ears, a nose, a mouth, a neck, shoulders, elbows, wrists,buttocks, knees, and ankles” as in the composition K1 in FIG. 21 on thebasis of the simplified model information acquired for the tappedsubject.

Then, in step S515 following step S514, the control unit 8 sets thetarget position Pt of tracking to a position corresponding to the fullshot.

Here, in this example, the control unit 8 uses, as the position Ps ofthe subject used in the tracking control processing, for example, aposition of a portion that fits within the screen 10 a when it is thecomposition K3 that is a composition in which the subject is zoomed inthe most, such as a position of a “neck” of the subject. Thus, it ispossible to prevent the position Ps of the subject from being lost andthe tracking from failing at the time of switching to the composition K2or the composition K3.

The target position Pt in step S515 is set to a position where at leastthe head of the subject (the upper end of the subject) does not frameout in the composition K1 with reference to the position Ps of thesubject, which is set to the position of the “neck” or the like asdescribed above.

In response to the setting of the target position Pt in step S515, thecontrol unit 8 increments the tap number identification value c by onein step S516, performs zooming control (control of the driver of thezoom lens in the driver unit 7) so as to achieve the target focal lengthin step S522, and returns to step S511.

Furthermore, if the tap number identification value c is one in stepS513, the control unit 8 proceeds to step S517 and calculates the focallength corresponding to the knee shot (composition K2) as the targetfocal length, and sets the target position Pt of tracking to theposition corresponding to the knee shot in step S518. Specifically, inthis example, the target position Pt is set to a position where at leastthe head of the subject does not frame out in the composition K2 withreference to the position Ps of the subject, which is set to theposition of the “neck” or the like as described above.

Then, in response to the execution of the processing in step S518, thecontrol unit 8 increments the tap number identification value c by onein step S516, then executes the zooming control in step S522, andreturns to step S511.

Furthermore, in a case where the tap number identification value c istwo in step S513, the control unit 8 proceeds to step S519 andcalculates the focal length corresponding to the bust shot (compositionK3) as the target focal length, and sets the target position Pt oftracking to the position corresponding to the bust shot in step S520.Specifically, in this example, the target position Pt is set to aposition where at least the head of the subject does not frame out inthe composition K3 with reference to the position Ps of the subject,which is set to the position of the “neck” or the like as describedabove.

Then, in response to the execution of the processing of step S520, thecontrol unit 8 resets the tap number identification value c to zero instep S521, then executes the zooming control of step S522, and returnsto step S511.

By the processing of steps S511 to S522 as described above, thecomposition K is switched in a toggling manner as compositionK1→composition K2→composition K3→composition K1→ . . . every time thetap operation is performed on the tapped subject.

Note that in the above description, the composition K is switchedaccording to the tap operation after the target detected subject ismoved to the screen center by the double tap operation, but this ismerely an example, and for example, the composition K can also beswitched according to another operation (including operations other thanthe tap operation) such as switching the composition K according to thesecond and subsequent tap operations of the target detected subject.

Furthermore, in the above description, the example has been described inwhich the adjustment of the composition is achieved by the change in theimaging direction and the change in the angle of view by the control ofthe platform device 50, but the adjustment of the composition can beachieved by changing at least one of the imaging direction or the angleof view.

Furthermore, the posture of the subject can be detected depending on thesimplified model information of the human body. In a case where theposture of the subject is detected, it is conceivable not to executeswitching of the composition K when the subject is in a predeterminedposture. For example, when the subject is in a squatting posture,avoiding changing of the composition K to the zoom-in side, or the likeis conceivable in consideration of the possibility of rising or jumpingthereafter.

3-2. Second Example

Next, a second example of the third embodiment will be described.

In the second example, the platform device 50 is controlled such that anangle-of-view center Pc is located on a front direction side of thesubject on the basis of direction information of the subject estimatedfrom the simplified model information.

FIGS. 25 and 26 are explanatory diagrams of a composition adjustmentmethod as a second example.

FIG. 25 illustrates an example in which a subject S3 facing the leftdirection and a subject S4 facing the front are displayed in the screen10 a. In the second example, after the touch operation on the detectedsubject, the composition is adjusted such that the angle-of-view centerPc is located on the front direction side of the detected subjectaccording to a second touch operation. In the examples of FIGS. 25 and26, the composition is adjusted such that the angle-of-view center Pc islocated on the front direction side of the subject S3 according to thesecond touch operation after the touch operation of the subject S3.

The flowchart of FIG. 27 illustrates a specific processing procedure forachieving the composition adjustment method as the second example.

Also in this case, the control unit 8 executes the processingillustrated in FIG. 13 as in the case of the first embodiment. However,the control unit 8 in this case executes the processing illustrated inFIG. 27 instead of the processing illustrated in FIG. 15 as the subjecttracking mode processing in step S107.

A difference from the processing illustrated in FIG. 15 is that spacingcontrol processing in step S601 is executed instead of the processing instep S308.

In the spacing control processing in step S601, the control unit 8estimates the direction of the detected subject (that is, the trackingtarget subject) for which the touch operation is detected in step S101from the simplified model information acquired from the imagerecognition processing unit 5. Then, the target position Pt is updatedsuch that the angle-of-view center Pc is located on the front directionside of the detected subject.

The control unit 8 advances the processing to step S309 in response tothe execution of the spacing control processing in step S601.

Through the processing as described above, the composition is adjustedsuch that the angle-of-view center Pc is located on the front directionside of the tracking target subject in response to the touch operationon the screen 10 a after the touch operation to designate the trackingtarget subject.

Since the adjustment is performed to a natural composition in which aspace is secured on the front side of the designated subject in responseto a simple operation of touching the screen 10 a, the adjustment can beperformed to a composition with less discomfort while reducing theoperation burden on the user.

Note that in the above description, an example has been described inwhich the spacing control processing in step S601 is executed inresponse to the touch operation on the screen 10 a after the touchoperation to designate the tracking target subject, but this is merelyan example, and the spacing control processing can also be executed inresponse to other operations (including operations other than the touchoperation).

4. Modification Examples

Here, the embodiment is not limited to the specific examples exemplifiedabove, and various modification examples are conceivable.

For example, in the above description, an example has been described inwhich the operation related to composition adjustment is performed as atouch operation on the screen 10 a, but instead of this, an operationusing a pointer such as a cursor displayed in the screen 10 a can beperformed. For example, the operation of designating the position of thesubject or the like may be an operation of pressing a predeterminedbutton or the like in a state where the pointer is positioned at adesired position. At this time, for example, an instruction on thedisplacement of the pointer in the screen 10 a can be given with adisplacement instruction control element such as a cross key.Furthermore, it is conceivable that the moving direction designationoperation related to adjustment of the imaging direction, adjustment ofthe in-screen position of the subject, and the like is, for example, anoperation of displacing the pointer by the above-described displacementinstruction control element in a state where a predetermined button ispressed.

Furthermore, in the above description, an example has been described inwhich the imaging device that captures an image displayed on the screen10 a and the imaging device that performs image-capturing for performingimage recognition such as subject detection are the common imagingdevice 1, but these imaging devices may be separate devices.

FIG. 28 illustrates an example thereof.

In this case, an imaging device 1A corresponds to an imaging device thatcaptures an image displayed on the screen 10 a, and the imaging device30 corresponds to an imaging device that performs imaging to performimage recognition. The imaging device 30 is, in other words, an imagingdevice separate from the imaging device in which the captured image isdisplayed on the screen 10 a.

As illustrated, the imaging device 30 and the imaging device 1A aremounted on the common platform device 50, and the imaging device 30 andthe imaging device 1A move in conjunction in the pan direction and thetilt direction according to a pan and tilt operation of the platformdevice 50.

FIG. 29 is a block diagram illustrating an internal configurationexample of the imaging device 30 and the imaging device 1A.

The imaging device 30 includes a lens system 2, an imaging unit 3, acamera signal processing unit 4, and an image recognition processingunit 5. Note that these units have already been described, and thusredundant description will be avoided.

The imaging device 1A is different from the imaging device 1 in that theimage recognition processing unit 5 is omitted.

In this case, the control unit 8 in the imaging device 1A executesprocessing (processing in FIGS. 13 to 15, 16, 17, 19, 23, 24, 27, andthe like) for composition adjustment similar to that of the first,second, or third embodiment, including the subject tracking controlprocessing, on the basis of the subject information Is acquired from theimage recognition processing unit 5 of the imaging device 30.

Note that in the imaging system having the configuration describedabove, a parallax or a difference in angle of view occurs between theimaging device 30 and the imaging device 1A, and thus the subjectposition is aligned between the captured images on the basis of theparallax or the difference in angle of view.

By employing the configuration of the imaging system as described above,the camera setting of the imaging device 1A can be a camera settingintended for drawing, and the camera setting of the imaging device 30can be a camera setting suitable for image recognition. For example,when panning of a subject is performed, the imaging device 1A performsimaging by reducing the shutter speed and setting appropriate exposure,but the imaging device 30 can be set to a camera setting different fromthis. That is, it is possible to robustly recognize a subject moving ata high speed in a state where the shutter speed is increased and the ISOsensitivity is increased.

Furthermore, the angle of view of the imaging device 30 is not affectedby the angle of view of the imaging device 1A, by making the angle ofview of the imaging device 30 wider than the angle of view of theimaging device 1A, the tracking target subject is less likely to beframed out, and the robustness of subject recognition can be increased.

Moreover, by increasing the frame rate of the imaging device 30 ascompared with the imaging device 1, the tracking performance of thesubject can be increased.

Furthermore, in the above description, the platform device capable ofdriving only in the pan direction and the tilt direction has beenexemplified as the platform device 50, but a platform device capable ofdriving in a roll direction (direction of arrow Dr in the drawings) likea platform device 50A exemplified in FIG. 30 can also be used.

In a case where the driving in the roll direction is enabled in thismanner, it is conceivable to adjust the direction of the imaging device1 in the roll direction according to the tracing operation on the screen10 a.

For example, FIG. 31 illustrates that an operation of touching thesubject S2 with the first finger (an operation of designating a trackingtarget subject) is performed, and then an operation of tracing thescreen 10 a in an arc shape with the second finger is performed. Inresponse to such an operation, as illustrated in FIG. 32, the controlunit 8 controls the platform device 50 such that the direction of theimaging device 1 in the roll direction changes by an amountcorresponding to the amount traced in the traced direction.

Here, in this case, in order to be distinguishable from the directionchange instruction operation in the roll direction, the direction changeinstruction operation in the pan direction and the tilt direction is anoperation of linearly tracing the second finger.

Note that in FIGS. 31 and 32, although the example has been described inwhich the direction of the imaging device 1 in the roll direction isadjusted according to the tracing operation by the second finger afterthe first finger touches the detected subject, such adjustment of thedirection in the roll direction is similarly performed even in a casewhere the first finger touches a subject other than the detectedsubject.

Furthermore, in a case where the simplified model information of thesubject is available as in the third embodiment, in response todetection that the target subject appears obliquely from the simplifiedmodel information, the direction in the roll direction can be adjustedsuch that the target subject is in an upright state in the screen 10 a(or in order to reduce the inclination).

5. Summary of Embodiment

As described above, the composition control device (Imaging device 1,1A) as an embodiment includes a control unit (8) configured to acquirecomposition designation operation information that is operationinformation designating a composition of an imaging device and includesinformation of a designated position on a screen (10 a) that displays acaptured image of the imaging device, and control an imaging range ofthe imaging device to adjust the composition of the imaging device onthe basis of subject information corresponding to the designatedposition of the composition designation operation information.

The composition designation operation information includes informationof the designated position on the screen on which the captured image isdisplayed. Thus, adjustment to a target composition is performed on thebasis of an intuitive operation of designating a position on the screenfor displaying the captured image. Specifically, in the configurationdescribed above, the composition of the imaging device is adjusted onthe basis of the subject information corresponding to the designatedposition on the screen. The subject information is information regardinga subject in the captured image, and is at least information indicatingthe presence or absence of the detected subject. Thus, the compositionis adjusted on the basis of an intuitive operation such as an operationof designating the position of the detected subject on the screen.

Therefore, operability can be improved for an operation related to thecomposition adjustment.

Furthermore, in the composition control device as the embodiment, in acase where the subject information indicates a detected subject detectedfrom the captured image, the control unit controls the imaging range insuch a manner as to track the detected subject.

Thus, an operation of designating the subject to be tracked is anoperation of designating the detected subject in the screen.

Therefore, operability can be improved for the operation related to thecomposition adjustment.

Moreover, in the composition control device as the embodiment, in a casewhere the subject information does not indicate the detected subject,the control unit does not perform control for tracking the subject atthe designated position.

Thus, for example, in a case where a subject estimated as not desired bythe user to be tracked, such as a background portion of the detectedsubject, is designated, tracking of the subject at the designatedposition is not performed.

Therefore, it is possible to achieve appropriate tracking controlaccording to an intention of the user.

Furthermore, in the composition control device as the embodiment, thecomposition designation operation information includes directiondesignation operation information based on a direction designationoperation that designates a direction in a plane of the screen, and thecontrol unit controls the imaging range on the basis of the directiondesignation operation information.

With this direction designation operation, it is possible to designate adirection related to the composition adjustment such as a direction ofchanging a position of the subject designated by an operation ofdesignating a position in the screen.

An operation of designating a direction related to the compositionadjustment is achieved by an intuitive operation on the screen, andoperability can be improved for the operation related to the compositionadjustment.

Furthermore, in the composition control device as the embodiment, thecomposition designation operation information is detected by a touchoperation on the screen, and the designated position is a touch startposition of the screen.

Thus, an operation of designating the subject to be tracked can be asimple operation of touching the subject on the screen.

Therefore, operability can be improved for the operation related to thecomposition adjustment.

Moreover, in the composition control device as the embodiment, thedirection designation operation information is detected by an operationof tracing the screen.

Thus, the operation of designating a direction related to thecomposition adjustment can be achieved by an intuitive operation ofsliding a finger in a direction in which the composition is desired tobe changed on the screen.

Therefore, operability can be improved for the operation related to thecomposition adjustment.

Furthermore, in a case where the operation of designating the positionis a touch operation and the tracking target subject is designated bythe operation of designating the position, designation of the trackingtarget subject and designation of movement of the position of thedesignated subject in the screen can be performed by an operation oftracing the screen in an arbitrary direction with the touched finger.

Therefore, ease of operation is increased, and operability can befurther improved.

Furthermore, in the composition control device as the embodiment, thecontrol unit controls the imaging range by controlling a platform device(50, 50A) to which the imaging device is attached.

Thus, it is possible to adjust the composition by changing the imagingdirection of the imaging device by the platform device.

Therefore, the degree of freedom of the composition adjustment can beincreased.

Furthermore, in the composition control device as the embodiment, theplatform device is capable of adjusting a direction of the imagingdevice in a pan or tilt direction, and the composition designationoperation information designating the pan or tilt direction is detectedby an operation of linearly tracing the screen.

Thus, the operation of designating the pan or tilt direction is achievedby an intuitive operation of linearly sliding the finger along the panor tilt direction.

Therefore, operability can be improved for the operation related to thecomposition adjustment.

Furthermore, in the composition control device as the embodiment, theplatform device is capable of adjusting a direction of the imagingdevice in a roll direction, and the composition designation operationinformation designating the roll direction is an operation of tracingthe screen in an arc shape.

Thus, an operation of designating the roll direction is achieved by anintuitive operation of sliding the finger in an arc shape along the rolldirection.

Therefore, operability can be improved for the operation related to thecomposition adjustment.

Moreover, in the composition control device as the embodiment, thecontrol unit controls the imaging range by controlling a platform deviceto which the imaging device is attached, and controls the platformdevice to change a direction of the imaging device in a directionopposite to the direction designated by the direction designationoperation in a case where the subject information indicates a detectedsubject detected from the captured image, and to change the direction ofthe imaging device in the direction designated by the directiondesignation operation in a case where the subject information does notindicate the detected subject.

In a case where a position designation operation is an operation ofdesignating the position of the detected subject, it can be estimatedthat the user desires to change the position of the detected subject inthe screen by a subsequent direction designation operation, and thus thedirection of the imaging device is changed in the direction opposite tothe direction designated by the direction designation operation. On theother hand, in a case where the position designation operation is anoperation of designating a position other than the detected subject, itcan be estimated that the user directly designates a changing directionof the imaging direction by the subsequent direction designationoperation, and thus the direction of the imaging device is changed inthe direction designated by the direction designation operation.

Thus, it is possible to achieve appropriate composition adjustmentreflecting an intention of the user.

Furthermore, in the composition control device as the embodiment, thecontrol unit controls the imaging range by controlling a platform deviceto which the imaging device is attached, and controls, in a case wherethe subject information indicates the detected subject, the platformdevice using a parameter corresponding to a type of the detected subjectas a parameter related to the tracking.

Thus, it is possible to perform tracking control by a parametercorresponding to the designated subject, such as performing trackingcontrol by a parameter with increased followability in a case where thedesignated subject is a type of subject that moves quickly andperforming tracking control by a parameter with decreased followabilityin a case where the designated subject is a type of subject that movesslowly.

Therefore, accuracy of the tracking control can be improved.

Furthermore, in the composition control device as the embodiment, theplatform device is capable of adjusting directions of the imaging devicein a pan direction and a tilt direction, and the control unit uses theparameter corresponding to the type of the detected subject for each ofcontrol in the pan direction and control in the tilt direction of theplatform device.

Thus, tracking characteristics in the pan direction and the tiltdirection can be individually set according to the type of thedesignated subject. It is possible that, for example, in a case wherethe subject is a bird, tracking responsiveness is increased in both thepan and tilt directions, and in a case where the subject is a dog, thetracking responsiveness in the pan direction is enhanced but thetracking responsiveness in the tilt direction is lowered.

It is possible to prevent tracking with excessively increasedresponsiveness from being performed in a direction in which the trackingresponsiveness does not need to be increased according to the type ofsubject, and to improve stability of the tracking control.

Moreover, in the composition control device as the embodiment, thecontrol unit performs tracking control in one of a first tracking modein which the parameter (the first parameter described above) having hightracking responsiveness in the pan direction and the tilt direction isused, a second tracking mode in which the parameter (the secondparameter described above) having low tracking responsiveness in the pandirection and the tilt direction is used, and a third tracking mode inwhich the parameter (the third parameter described above) having lowtracking responsiveness in one of the pan direction and the tiltdirection and having high tracking responsiveness in the other of thepan direction and the tilt direction according to the type of thedetected subject.

Thus, it is possible to prevent tracking in which responsiveness isexcessively enhanced in a direction in which it is not necessary toenhance the tracking responsiveness according to the type of thedesignated subject.

Therefore, the stability of the tracking control can be improved.

Furthermore, in the composition control device as the embodiment, thecontrol unit adjusts, after control of the platform device using theparameter corresponding to the type of the detected subject is started,the parameter on the basis of an error between a position of thedetected subject and a target position of tracking.

It is possible to determine whether or not the set parameter is suitablefor actual movement of the subject according to the magnitude of theerror between a designated position of the subject and the targetposition of tracking.

Therefore, by adjusting the parameter on the basis of the error, theparameter can be adjusted to a parameter suitable for actual movement ofthe subject, and the stability of the tracking control can be improved.

Furthermore, in the composition control device as the embodiment, thecontrol unit controls, in a case where the subject information indicatesa detected subject detected as a person in the screen, the imaging rangeon the basis of simplified model information of a human body detectedfor the detected subject.

The simplified model information of the human body is informationindicating an arrangement state of a plurality of parts constituting thehuman body such as eyes, ears, a nose, a mouth, a neck, shoulders,elbows, wrists, buttocks, knees, and ankles in the human body. With theabove configuration, the control of the imaging range for changing thecomposition can be performed not on the basis of the operation ofdirectly instructing the driving direction or the drive amount of thecamera platform or the focal length adjustment operation, but on thebasis of a position and posture of each part of the subject, forexample, estimated from the simplified model information describedabove.

Therefore, it is possible to reduce the operation burden on the userrelated to the composition adjustment.

Furthermore, by using the simplified model information, estimationaccuracy of a specific portion such as the whole body part or a halfbody part of the subject to be contained in the screen (in the capturedimage) is increased, and thus precision of the composition adjustmentcan be improved. For example, although a specific portion such as thewhole body or a half body of the subject can be estimated from a faceportion detected by face detection, the estimation accuracy of thespecific portion is higher than that in that case, and the precision ofthe composition adjustment can be improved.

Furthermore, in the composition control device as the embodiment, thecontrol unit controls the imaging range in such a manner that a centerof an angle of view is located on a front direction side of the detectedsubject on the basis of information of a direction of the detectedsubject estimated from the simplified model information.

Thus, it is possible to adjust the composition to a natural compositionin which a space is secured on a front side of the designated subject.

Therefore, it is possible to adjust the composition with less discomfortwhile reducing the operation burden on the user.

Furthermore, in the composition control device as the embodiment, thecomposition designation operation information is detected by a touchoperation on the screen, the designated position is a touch startposition of the screen, and the control unit controls the imaging rangebased on the simplified model information on the basis of an operationon the screen after detection of the designated position.

Thus, the composition adjustment based on the simplified modelinformation, such as adjustment to a composition in which the entirebody of the subject appears or a composition in which only a half bodyappears for example, can be achieved by a simple operation ofdesignating a target subject by a touch operation on the screen and thenperforming an operation on the screen.

Therefore, an operation related to the composition adjustment can befacilitated, and the operation burden on the user can be reduced.

Moreover, in the composition control device as the embodiment, thecontrol unit controls the imaging range on the basis of a result ofimage recognition processing performed on a captured image by an imagingdevice (the same 30) separate from the imaging device (identical 1, 1A)in which a captured image is displayed on the screen.

Thus, as a captured image used for the image recognition processing, acaptured image with camera settings suitable for the image recognitionprocessing can be used instead of the captured image with camerasettings according to a drawing intention of the user.

Therefore, in a case where imaging range control for the compositionadjustment is performed on the basis of the result of the imagerecognition processing, accuracy of the image recognition processing canbe improved, and accuracy of the composition adjustment can be improved.

Furthermore, a composition control method as an embodiment is acomposition control method including acquiring composition designationoperation information that is operation information designating acomposition of an imaging device and includes information of adesignated position on a screen that displays a captured image of theimaging device, and controlling an imaging range of the imaging deviceto adjust the composition of the imaging device on the basis of subjectinformation corresponding to the designated position of the compositiondesignation operation information.

Even with such a composition control method as an embodiment, a similaroperation and effect to those of the composition control device as theembodiment described above can be obtained.

A program according to an embodiment is a program causing an informationprocessing device to achieve a function including acquiring compositiondesignation operation information that is operation informationdesignating a composition of an imaging device and includes informationof a designated position on a screen that displays a captured image ofthe imaging device, and controlling an imaging range of the imagingdevice to adjust the composition of the imaging device on the basis ofsubject information corresponding to the designated position of thecomposition designation operation information.

That is, the program causes the information processing device to executethe processing described in FIGS. 13 to 15, 16, 17, 19, 23, 24, 27, andthe like.

With such a program, it is easy to achieve the composition controldevice as the embodiment.

Then, such a program can be stored in advance in a recording mediumbuilt in a device such as a computer device, a ROM in a microcomputerhaving a CPU, or the like. Alternatively, the program can be temporarilyor permanently stored in a removable recording medium such as asemiconductor memory, a memory card, an optical disk, a magneto-opticaldisk, or a magnetic disk. Such a removable recording medium can also beprovided as what is called package software.

Furthermore, such a program can be installed from the removablerecording medium into a personal computer or the like, or can bedownloaded from a download site via a network such as a local areanetwork (LAN) or the Internet.

Note that effects described in the present description are merelyexamples and are not limited, and other effects may be provided.

6. Present Technology

Note that the present technology can employ configurations as follows.

(1)

A composition control device including:

a control unit configured to acquire composition designation operationinformation that is operation information designating a composition ofan imaging device and includes information of a designated position on ascreen that displays a captured image of the imaging device, and controlan imaging range of the imaging device to adjust the composition of theimaging device on the basis of subject information corresponding to thedesignated position of the composition designation operationinformation.

(2)

The composition control device according to (1) above, in which

in a case where the subject information indicates a detected subjectdetected from the captured image,

the control unit controls the imaging range in such a manner as to trackthe detected subject.

(3)

The composition control device according to (2) above, in which

in a case where the subject information does not indicate the detectedsubject,

the control unit does not perform control for tracking the subject atthe designated position.

(4)

The composition control device according to any one of (1) to (3) above,in which

the composition designation operation information includes directiondesignation operation information based on a direction designationoperation that designates a direction in a plane of the screen, and

the control unit controls the imaging range on the basis of thedirection designation operation information.

(5)

The composition control device according to any one of (2) to (4) above,in which

the composition designation operation information is detected by a touchoperation on the screen, and

the designated position is a touch start position of the screen.

(6)

The composition control device according to (4) above, in which

the direction designation operation information is detected by anoperation of tracing the screen.

(7)

The composition control device according to any one of (1) to (6) above,in which

the control unit controls the imaging range by controlling a platformdevice to which the imaging device is attached.

(8)

The composition control device according to (7) above, in which

the platform device is capable of adjusting a direction of the imagingdevice in a pan or tilt direction, and

the composition designation operation information designating the pan ortilt direction is detected by an operation of linearly tracing thescreen.

(9)

The composition control device according to (7) or (8) above, in which

the platform device is capable of adjusting a direction of the imagingdevice in a roll direction, and

the composition designation operation information designating the rolldirection is an operation of tracing the screen in an arc shape.

(10)

The composition control device according to any one of (4) to (9) above,in which

the control unit

controls the imaging range by controlling a platform device to which theimaging device is attached, and

controls the platform device to change a direction of the imaging devicein a direction opposite to the direction designated by the directiondesignation operation in a case where the subject information indicatesa detected subject detected from the captured image, and to change thedirection of the imaging device in the direction designated by thedirection designation operation in a case where the subject informationdoes not indicate the detected subject.

(11)

The composition control device according to any one of (2) to (10)above, in which

the control unit

controls the imaging range by controlling a platform device to which theimaging device is attached, and

controls, in a case where the subject information indicates the detectedsubject, the platform device using a parameter corresponding to a typeof the detected subject as a parameter related to the tracking.

(12)

The composition control device according to (11) above, in which

the platform device is capable of adjusting directions of the imagingdevice in a pan direction and a tilt direction, and

the control unit uses the parameter corresponding to the type of thedetected subject for each of control in the pan direction and control inthe tilt direction of the platform device.

(13)

The composition control device according to (12) above, in which

the control unit performs tracking control in one of a first trackingmode in which the parameter having high tracking responsiveness in thepan direction and the tilt direction is used, a second tracking mode inwhich the parameter having low tracking responsiveness in the pandirection and the tilt direction is used, and a third tracking mode inwhich the parameter having low tracking responsiveness in one of the pandirection and the tilt direction and having high tracking responsivenessin the other of the pan direction and the tilt direction according tothe type of the detected subject.

(14)

The composition control device according to any one of (11) to (13)above, in which

the control unit adjusts, after control of the platform device using theparameter corresponding to the type of the detected subject is started,the parameter on the basis of an error between a position of thedetected subject and a target position of tracking.

(15)

The composition control device according to any one of (1) to (14)above, in which

the control unit controls, in a case where the subject informationindicates a detected subject detected as a person in the screen, theimaging range on the basis of simplified model information of a humanbody detected for the detected subject.

(16)

The composition control device according to (15) above, in which

the control unit controls the imaging range in such a manner that acenter of an angle of view is located on a front direction side of thedetected subject on the basis of information of a direction of thedetected subject estimated from the simplified model information.

(17)

The composition control device according to (15) or (16) above, in which

the composition designation operation information is detected by a touchoperation on the screen,

the designated position is a touch start position of the screen, and

the control unit controls the imaging range based on the simplifiedmodel information on the basis of an operation on the screen afterdetection of the designated position.

(18)

The composition control device according to any one of (1) to (17)above, in which

the control unit controls the imaging range on the basis of a result ofimage recognition processing performed on a captured image by an imagingdevice separate from the imaging device in which a captured image isdisplayed on the screen.

REFERENCE SIGNS LIST

-   1, 1A Imaging device-   5 Image recognition processing unit-   7 Driver unit-   8 Control unit-   9 Operation unit-   9 a Touch panel-   10 Display unit-   10 a Screen-   30 Imaging device-   50, 50A Platform device-   56 Actuator-   57 Drive control unit-   F1 Position error calculation processing unit-   F2 Control value calculation processing unit-   F3 Drive amount conversion processing unit-   Pt Target position-   d Error (position error)

1. A composition control device comprising: a control unit configured toacquire composition designation operation information that is operationinformation designating a composition of an imaging device and includesinformation of a designated position on a screen that displays acaptured image of the imaging device, and control an imaging range ofthe imaging device to adjust the composition of the imaging device on abasis of subject information corresponding to the designated position ofthe composition designation operation information.
 2. The compositioncontrol device according to claim 1, wherein in a case where the subjectinformation indicates a detected subject detected from the capturedimage, the control unit controls the imaging range in such a manner asto track the detected subject.
 3. The composition control deviceaccording to claim 2, wherein in a case where the subject informationdoes not indicate the detected subject, the control unit does notperform control for tracking the subject at the designated position. 4.The composition control device according to claim 1, wherein thecomposition designation operation information includes directiondesignation operation information based on a direction designationoperation that designates a direction in a plane of the screen, and thecontrol unit controls the imaging range on a basis of the directiondesignation operation information.
 5. The composition control deviceaccording to claim 2, wherein the composition designation operationinformation is detected by a touch operation on the screen, and thedesignated position is a touch start position of the screen.
 6. Thecomposition control device according to claim 4, wherein the directiondesignation operation information is detected by an operation of tracingthe screen.
 7. The composition control device according to claim 1,wherein the control unit controls the imaging range by controlling aplatform device to which the imaging device is attached.
 8. Thecomposition control device according to claim 7, wherein the platformdevice is capable of adjusting a direction of the imaging device in apan or tilt direction, and the composition designation operationinformation designating the pan or tilt direction is detected by anoperation of linearly tracing the screen.
 9. The composition controldevice according to claim 7, wherein the platform device is capable ofadjusting a direction of the imaging device in a roll direction, and thecomposition designation operation information designating the rolldirection is an operation of tracing the screen in an arc shape.
 10. Thecomposition control device according to claim 4, wherein the controlunit controls the imaging range by controlling a platform device towhich the imaging device is attached, and controls the platform deviceto change a direction of the imaging device in a direction opposite tothe direction designated by the direction designation operation in acase where the subject information indicates a detected subject detectedfrom the captured image, and to change the direction of the imagingdevice in the direction designated by the direction designationoperation in a case where the subject information does not indicate thedetected subject.
 11. The composition control device according to claim2, wherein the control unit controls the imaging range by controlling aplatform device to which the imaging device is attached, and controls,in a case where the subject information indicates the detected subject,the platform device using a parameter corresponding to a type of thedetected subject as a parameter related to the tracking.
 12. Thecomposition control device according to claim 11, wherein the platformdevice is capable of adjusting directions of the imaging device in a pandirection and a tilt direction, and the control unit uses the parametercorresponding to the type of the detected subject for each of control inthe pan direction and control in the tilt direction of the platformdevice.
 13. The composition control device according to claim 12,wherein the control unit performs tracking control in one of a firsttracking mode in which the parameter having high tracking responsivenessin the pan direction and the tilt direction is used, a second trackingmode in which the parameter having low tracking responsiveness in thepan direction and the tilt direction is used, and a third tracking modein which the parameter having low tracking responsiveness in one of thepan direction and the tilt direction and having high trackingresponsiveness in the other of the pan direction and the tilt directionaccording to the type of the detected subject.
 14. The compositioncontrol device according to claim 11, wherein the control unit adjusts,after control of the platform device using the parameter correspondingto the type of the detected subject is started, the parameter on a basisof an error between a position of the detected subject and a targetposition of tracking.
 15. The composition control device according toclaim 1, wherein the control unit controls, in a case where the subjectinformation indicates a detected subject detected as a person in thescreen, the imaging range on a basis of simplified model information ofa human body detected for the detected subject.
 16. The compositioncontrol device according to claim 15, wherein the control unit controlsthe imaging range in such a manner that a center of an angle of view islocated on a front direction side of the detected subject on a basis ofinformation of a direction of the detected subject estimated from thesimplified model information.
 17. The composition control deviceaccording to claim 15, wherein the composition designation operationinformation is detected by a touch operation on the screen, thedesignated position is a touch start position of the screen, and thecontrol unit controls the imaging range based on the simplified modelinformation on a basis of an operation on the screen after detection ofthe designated position.
 18. The composition control device according toclaim 1, wherein the control unit controls the imaging range on a basisof a result of image recognition processing performed on a capturedimage by an imaging device separate from the imaging device in which acaptured image is displayed on the screen.
 19. A composition controlmethod comprising: acquiring composition designation operationinformation that is operation information designating a composition ofan imaging device and includes information of a designated position on ascreen that displays a captured image of the imaging device, andcontrolling an imaging range of the imaging device to adjust thecomposition of the imaging device on a basis of subject informationcorresponding to the designated position of the composition designationoperation information.
 20. A program causing an information processingdevice to achieve a function comprising: acquiring compositiondesignation operation information that is operation informationdesignating a composition of an imaging device and includes informationof a designated position on a screen that displays a captured image ofthe imaging device, and controlling an imaging range of the imagingdevice to adjust the composition of the imaging device on a basis ofsubject information corresponding to the designated position of thecomposition designation operation information.